The consequential validity of ABFM examinations.

Measurement scholar Samuel Messick, defines validity as “an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores. . . ” (p. 13). Messick’s definition of validity differed from those of previous validity theorists in that he acknowledged that test scores often affect social policy and thus argued social consequences should be examined. Messick referred to this form of validity as “consequential validity.” Shepard further clarified social consequences to include both the positive/negative and intended/unintended consequences that may result from scorebased inferences. The purpose of this article is to discuss consequential validity as it pertains to American Board of Family Medicine (ABFM) examinations. To date, the ABFM has published numerous articles that evidence the adequacy and appropriateness of inferences based on examination scores. Many of these articles are validity studies that involve rigorous data analyses with state-ofthe-art psychometric methods, whereas others advocate responsible score reporting and interpretation. Given that Messick’s framework for validity also includes the social consequences that may result from score inferences, it is important to address this aspect of validity as well. Unlike other indicators of validity, consequential validity has less to do with data analysis and more to do with making inferences. Thus, the extent to which ABFM examination scores are appropriately interpreted and used depends largely on others. Our intention is to clarify some key inferences that should and should not be made about ABFM examination score results. ABFM examinations measure a physician’s fund of medical knowledge within the context of the clinical practice of the specialty of family medicine. The examinations do not measure other important aspects of family medicine, such as one’s clinical or procedural skills, the ability to communicate with patients, professional attitudes and behaviors, the ability to practice within a system of care, and the ability to learn from the practice of family medicine to continuously improve patient care. Unfortunately, many consumers of ABFM examination score results often make inappropriate inferences about what exactly the scores mean. For example, consumers rightly infer that a passing score conferring certification is a surrogate for quality. Consumers also rightly infer that a passing score and subsequent certification should facilitate privileges within a hospital setting or credentials within a medical group. Unfortunately, consumers sometimes wrongly infer that a nonpassing score indicates that a physician is not worthy of being certified, and thus by extension, does not or is not capable of providing high-quality care. In addition, some consumers incorrectly infer that a higher examination score is more indicative of a better physician (compared with a physician who has a lower score); it is well understood, however, that multiple factors determine whether a physician is “good.” It is critical that consumers understand that simply because a physician fails the Maintenance of Certification for Family Physicians (MC-FP) examination does not mean she or he is incapable of providing high-quality care or is incapable of becoming more knowledgeable about the important body of knowledge that defines the specialty of family medicine. Knowledge is fluid; thus everyone has the propensity to become more knowledgeable. In fact, over the years the ABFM staff has heard from hundreds of physicians who initially failed the MC-FP examination and who subsequently developed an improved study plan and passed on their Conflict of interest: The authors are from the ABFM.